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ABSTRACT 

In  this  paper,  we  discuss  a  neuroimaging  experiment  that  employed  a  mission-based  scenario 
(MBS)  design,  a  new  approach  for  designing  experiments  in  simulated  environments  for  human 
subjects  [1J.  This  approach  aims  to  enhance  the  realism  of  the  Soldier-task-environment 
interaction  by  eliminating  many  of  the  tightly-scripted  elements  of  a  typical  laboratory > 
experiment;  however,  the  absence  of  these  elements  introduces  severed  challenges  for  both  the 
experimental  design  and  statistical  analysis  of  the  experimented  data.  Here,  we  describe  an  MBS 
experiment  using  a  simulated,  closed-hatch  crewstation  environment.  For  each  experimented 
session,  two  Soldiers  participated  as  a  Commander-Driver  team  to  perform  six  simulated  low- 
threat  security  patrol  missions.  We  discuss  challenges  faced  while  designing  and  implementing  the 
experiment  before  addressing  analysis  approaches  appropriate  for  this  type  of  experimentation. 
We  conclude  by  highlighting  three  example  transition  pathways  from  MBS  experiments  to 
enhanced  Army  capabilities  using  a  class  of  neurotechnologies  called  Brain-Computer  Interaction 
Technologies. 


introduction 

Imagine  a  system  that  can  identify  operator  fatigue  during 
a  long-term  vigilance  task  and  automatically  execute  a 
fatigue  mitigation  strategy.  This  system  would  be  an 
example  of  a  Brain-Computer  Interaction  Technology 
(BCIT),  a  class  of  neurotechnologies,  that  aim  to  improve 
task  performance  by  incorporating  measures  of  brain  activity 
to  optimize  the  interactions  between  Soldiers  and  systems 
[2],  This  example  BCIT  would  use  real-time  processing  of 
the  system  operator’s  neural  data  to  identify  a  brain  state  that 
indicates  suboptimal  performance  in  the  absence  of  any 
behavioral  inaccuracies,  as  vigilance  does  not  require  overt 
responses.  If  research  has  identified  fatigue  mitigation 
strategies,  the  BCIT  could  automatically  execute  the 
mitigation  strategy  when  a  fatigue  state  is  detected.  This 
“automatic”  interaction  between  the  operator  and  the  system 


would  minimize  vigilance  decrements  based  on  operator 
fatigue.  This  is  just  one  example  to  illustrate  the  potential 
performance  enhancement  of  a  fatigue-based  BCIT  where 
neural  data  indicates  task-relevant  brain  states  in  the  absence 
of  overt  operator  behavior. 

The  goal  of  developing  BCITs  is  to  improve  the 
performance  of  healthy  individuals  working  in  complex, 
real-world  environments,  such  as  a  vehicle-based 
crewstation.  A  crewstation  environment  can  provide  a 
stepping  stone  between  the  simplified  task  and  laboratory 
environment  in  traditional  neuroscience  experiments  and  the 
complexities  of  dismounted  Soldiers  navigating  through  a 
rich  sensory  world.  The  advantage  of  a  transition  from  the 
laboratory  to  the  real-world  through  a  crewstation 
environment  is  especially  clear  when  studying  task 
performance  under  closed-hatch  operations,  where  the  only 
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view  of  the  external  world  from  inside  the  vehicle  is  through 
an  indirect  vision  system.  With  no  open  windows  on  the 
vehicle,  the  crewstation  environment  can  be  precisely 
described  and  monitored  based  on  the  settings  and  logs  of 
the  vehicle  systems.  Thus,  the  crewstation  is  more 
constrained  than  a  dismounted  environment  where  it  is  much 
harder  to  monitor  the  events  in  the  external  world  or  know 
what  sensory  information  about  that  world  is  available  to  the 
Soldier. 

Of  course,  a  crewstation  environment  has  many  substantial 
differences  from  a  typical,  neuroscience  laboratory.  A 
traditional  neuroscience  experiment  is  conducted  in  a  quiet, 
barren  room  where  stimuli  are  presented  in  isolation  on  a 
single  computer  monitor,  and  participants  are  asked  to 
minimize  both  body  movements  and  eye  blinks  in  order  to 
minimize  artifacts  in  the  physiological  measurements.  While 
this  environment  is  designed  for  studying  specific  cognitive 
processes,  without  confounds  of  concurrent  tasks  or 
excessive  noise  overriding  the  physiological  signal  of 
interest,  the  laboratory  may  limit  our  understanding  about 
how  tasks  are  performed  in  the  real-world,  where  our  bodies 
and  eyes  move  freely  and  we  are  often  faced  with  multiple, 
concurrent  tasks.  The  crewstation  environment  is  one  way  to 
transition  from  the  constrained,  simplified  laboratory  to 
more  complex  environments  where  the  sensory  information 
is  richer  and  more  realistic  than  the  lab,  yet  still  definable 
based  on  the  vehicle  systems  that  enable  closed-hatch 
operations. 

A  simulated  crewstation  environment  provides  the  ability 
to  extract  multiple  measures  of  a  participant’s  interactions 
with  the  environment  [3].  The  sensory  experience  of  the 
closed  hatch  environment  can  be  captured  based  on  system 
logs  (crewstation  state  changes,  screen  configurations, 
sensor  settings),  and  interactions  with  these  systems  provide 
behavioral  measures  (e.g.,  reaction  times,  accuracy  of  button 
presses).  Eye -tracking  measures  capture  where  the 
participant  is  looking,  while  physiological  measures  (e.g., 
heart  rate,  galvanic  skin  response)  can  capture  information 
about  a  participant’s  internal  states  (e.g.,  stress).  This 
crewstation  environment  is  also  amenable  for  portable 
neuroimaging  techniques  (e.g.,  Electroencephalography 
(EEG),  Functional  Near-Infrared  Spectroscopy  (fNIRS))  to 
collect  brain  data  during  the  execution  of  tasks  to  understand 
how  the  brain  functions  in  this  complex  setting.  These 
multiple  measurements  provide  an  avenue  to  record, 
analyze,  and  understand  human  behavior  in  more  realistic 
settings. 

The  underlying  argument  here  is  that  to  develop  BC1T  for 
a  real-world  crewstation,  the  neuroscience  experiment  needs 
to  match  the  real-world  environment  as  accurately  as 
possible.  The  need  for  this  fidelity  occurs  on  many  levels. 
The  brain  measures  needed  to  augment  a  BCIT  must  be 
robust  to  the  increased  signal  artifacts  based  on  body  and 


eye  movements  in  complex  environments  compared  to 
constrained  movements  in  laboratory  environments.  The 
physiological  signatures  of  the  task  may  change  when  a 
person  performs  multiple  tasks  in  the  crewstation,  compared 
to  single,  isolated  tasks  in  the  laboratory.  The  physiological 
signatures  may  change  based  on  the  sensory  complexity  of  a 
crewstation  compared  to  the  white,  barren  walls  of  a 
laboratory.  With  finite  resources,  the  BCIT  development 
approach  that  is  most  likely  to  succeed  seems  to  require 
experimentation  in  a  targeted  environment  in  order  to 
minimize  differences  between  development  and 
implementation.  This  shift  from  a  laboratory  to  a  crewstation 
necessitates  changes  in  both  the  experimental  designs  used 
to  simulate  a  more  realistic  complexity  of  the  tasks  and  the 
data  analyses  employed  to  account  for  the  decrease  in  task 
constraints  and  signal  to  noise  ratio.  Consequently,  scientists 
at  the  Army  Research  Laboratory  (ARL)  are  developing  the 
Mission-Based  Scenario  (MBS)  experimental  design  concept 
to  balance  the  needs  for  constraining  experimental 
environments  (i.e.,  minimize  confounds,  make  data  analysis 
tractable)  and  the  need  for  sensory  and  task  complexity  (i.e., 
better  emulate  real-world  environments). 

Mission-Based  Scenario  (MBS)  Design  &  Analysis 

To  invoke  more  realistic  responses  in  a  simulated 
environment,  an  MBS  experiment  must  ensure  that  the 
simulated  environment  does  not  actively  disrupt  the 
participant’s  behavior,  either  through  surprise,  unusual 
occurrences  or  forced,  unnatural  behavior.  These  types  of 
surprises  result  in  a  breakage  in  the  Soldier’s  sense  of 
presence  or  suspension  of  disbelief  in  the  environment  [4]. 
Presence  can  be  viewed  as  the  extent  to  which  a  user  feels 
physically  present  within  a  virtual  environment  [5],  while 
suspension  of  disbelief  describes  the  extent  to  which  an 
individual  becomes  mentally  engrossed  in  the  environment 
[4].  If  the  participant’s  presence  and  suspension  of  disbelief 
are  regularly  disrupted  by  the  experimental  environment,  it 
is  difficult  to  argue  that  the  resulting  studied  behavior  will 
be  in  any  way  natural  [6],  The  MBS  concept  aims  to  address 
this  issue  in  three  ways:  1)  to  situate  experiments  within  a 
more  realistic  mission  context;  2)  to  incorporate  tasks,  task 
loadings,  and  environmental  interactions  that  are  consistent 
with  the  mission’s  operational  context;  and  3)  to  permit 
multiple  sequences  of  actions/tasks  to  complete  mission 
objectives  [1],  In  short,  an  experiment  employing  a  mission- 
based  scenario  design  aims  to  simulate  a  more  realistic 
mission  experience  in  order  to  capture  the  relevant 
contextual  effects  on  the  participant’s  tasks,  actions,  and 
environmental  interactions. 

MBS  experimental  designs  demand  different  data  analysis 
approaches.  Traditional  neuroimaging  experiments  in  a 
laboratory  isolate  task  processing  by  time-locking  to  the 
period  of  time  (experimental  trial)  when  a  single  task  occurs; 
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however,  in  simulated  missions,  the  timing  of  events  and 
tasks  differ  with  more  variability.  Consequently,  the  analysis 
approach  for  a  mission-based  scenario  design  must  provide  a 
way  to  capture  task-relevant  features  in  the  EEG  signal 
despite  the  complexity  of  a  less  constrained  scenario  design. 
Although  traditional,  event-related  analyses  could  still  be 
used  for  an  MBS  experiment,  this  paper  will  emphasize 
analyses  that  employ  a  pattern  classification  analysis 
approach.  These  classification  examples  aim  to  identify 
features  of  EEG  dynamics  that  reflect  global  patterns  that 
can  predict  particular  mission  tasks  or  relevant  brain  states 
that  impact  task  performance.  These  task-relevant  features 
could  then  be  used  in  the  design  of  BCIT  neurotechnologies 
that  allow  systems  to  adapt  to  the  dynamic  changes  in  a 
user’s  brain  state  to  enhance  performance  [2], 


Figure  1:  Soldier  sitting  on  a  six  degree-of- freedom  Ride 
Motion  Simulator  to  simulate  realistic  movements  of  a 
vehicle.  Photo  provided  by  Detroit  Arsenal  Media  Services. 


Paper  Overview 

In  this  paper,  we  discuss  a  mission-based  simulation 
experiment  looking  at  a  Commander-Driver  team  that  aimed 
to  link  behavioral  task  performance  during  low-threat 
security  patrol  missions  with  neural  data  captured  by  scalp 
EEG  sensors.  The  study  emphasizes  the  role  of  a  Vehicle 
Commander.  In  security  patrol  missions,  the  Commander  is 
responsible  for  many  tasks,  including  route  planning  and 
navigation,  responding  to  various  auditory  communications 
about  mission  status  and  coordination,  and  maintaining  local 
situational  awareness  (LSA)  to  detect  and  report  targets.  The 
frequency  and  difficulty  level  of  these  numerous  tasks  will 
vary  throughout  the  mission,  and  it  is  likely  that  the 
Commander  will  often  have  to  manage  many  of  these  tasks 
concurrently. 


With  our  initial  research  emphasis  on  MBS  for  crewstation 
environments,  this  experiment  was  conducted  on  a  Ride 
Motion  Simulator  (RMS,  Figure  1)  that  simulates  as 
realistically  as  possible  the  movements  of  vehicles  at  the 
Tank  Automotive  Research,  Development  and  Engineering 
Center’s  (TARDEC)  Ground  Vehicle  Simulation  Laboratory 
(GVSL)  at  the  Detroit  Arsenal  in  Warren,  ME  Thus,  this 
experiment  emphasizes  another  layer  of  transition  from  the 
laboratory  to  the  real-world  -  before  moving  the  MBS  to  a 
field  experiment  with  vehicles  navigating  around 
unconstrained  terrain,  the  transition  begins  with  a  simulated, 
mobile  crewstation  environment  to  gradually  move  from  a 
fully  constrained,  stationary  laboratory  environment  to  the 
dynamics  of  a  field  test. 

After  providing  an  overview  of  the  simulated  mission 
environment,  we  discuss  the  design  challenges  and  decisions 
stemming  from  the  three  aims  of  the  MBS  concept.  Analyses 
of  this  dataset  are  underway  [7],  but  instead  of  describing  a 
specific  result,  this  paper  discusses  three  example  pathways 
to  characterize  how  MBS  experiments  foster  the 
development  of  BCITs,  and  how  these  neurotechnologies 
show  the  potential  for  neuroscience  experiments  to  enhance 
Army  capabilities. 

MISSION-BASED  SCENARIO  EXPERIMENT 

This  experiment  was  performed  using  a  simulated 
crewstation  environment  for  a  Vehicle  Commander  and 
Driver  during  a  series  of  low-threat  security  patrol  missions. 
With  space  for  only  one  crew  member  on  the  RMS  (Figure 
1),  we  focused  on  the  Commander  and  his  tasks.  The 
experimental  objective  was  to  study  brain  dynamics  of  the 
Commander  during  as  realistic  as  possible  task  performance. 

Experiment  Overview 

Each  week  for  seven  weeks,  two  Army  Sergeants  were 
flown  in  from  U.S.  Army  Maneuver  Center  of  Excellence  at 
Fort  Knox,  KY  and  Fort  Benning,  GA.  All  of  the  male 
participants  had  a  Military  Occupational  Specialty  (MOS) 
1  IB  (Infantryman),  MOS  19D  (Cavalry  Scout),  or  MOS 
19K  (Armor  Crewman).  The  Soldiers  were  all  combat 
veterans  of  Iraq  or  Afghanistan.  Except  for  the  final  week, 
the  Soldiers  did  not  know  each  other  before  arriving  for  the 
experiment,  and  in  all  cases,  the  Soldiers  had  not  been 
deployed  together.  After  completing  a  day  of  training,  the 
participants  completed  two  days  of  experimentation.  On  day 
one,  one  Soldier  served  as  Commander  and  the  other  as 
Driver,  and  on  day  two,  the  Soldiers  swapped  roles. 
Consequently,  a  total  of  14  Commander-Driver  datasets 
were  collected  across  the  seven  weeks  of  data  collection 
between  September  and  November  2010. 

Each  team  pair  completed  six,  low-threat  security  patrol 
missions  through  a  simulated  urban  desert  terrain.  The 
Commander  sat  on  a  six  degree-of-freedom,  ride  motion 
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simulator  using  two  touchscreen  interfaces  to  interact  with 
the  environment  through  a  360  degree  LSA  system  (Figure 
2).  The  Commander  also  had  a  digital  map  with  real-time 
information  about  vehicle  location,  a  printed  map  to  his  right 
with  the  mission  checkpoints  and  his  hand-drawn  mission 
route,  and  a  touchscreen  button  interface  for  radio-related 
tasks.  The  Driver  sat  on  a  static  platform  using  a  single 
monitor  with  only  a  60  degree  horizontal,  straight  ahead 
view  of  the  environment  (Figure  3)  and  no  map.  The 
Driver’s  steering  wheel  controlled  the  movements  of  the 
RMS,  allowing  him  to  follow  the  Commander’s  directions  to 
maneuver  the  vehicle  through  the  simulated  mission.  The 
Driver  was  not  instrumented  for  any  physiological 
monitoring,  while  both  eye-tracking  and  EEG  data,  which 
reflects  brain  activity  through  measurement  of  the  pattern  of 
electrical  activity  at  the  scalp,  were  collected  on  the 
Commander.  Even  though  the  two  Soldiers  were  not  on  the 
same  platform,  the  two  communicated  with  one  another 
through  a  two-way  radio  headset,  and  they  also  heard 
additional,  pre-recorded  audio  communication  from  other 
simulated  agents  in  the  environment  through  a  separate  set 
of  loudspeakers.  One  of  the  pre-recorded  voices  was  a 
simulated  tactical  operating  commander  (TOC)  who 
provided  the  mission  directives  a  Commander  would  expect 
on  a  patrol  mission.  One  of  the  experimenters  also  operated 
a  soundboard  with  controls  to  activate  pre-recorded  TOC 
responses,  facilitating  simulated  interactions  between  the 
TOC  and  Commander;  for  example,  one  button  allowed  the 
TOC  to  respond  “Roger”  when  the  Commander  called  in 
mission  reports. 


Figure  2:  Vehicle  Commander  interacting  with  the 
simulated  mission  environment,  wearing  a  64  channel  EEG 
cap.  Photo  provided  by  Detroit  Arsenal  Media  Services. 

For  each  mission,  the  pair  drove  from  a  Forward  Operating 
Base  (FOB)  to  their  patrol  sector  in  the  city,  Desert  Metro. 
The  Commander  then  navigated  through  a  simulated  city  to 
reach  three  security  checkpoints  at  specified  times  for 
mission-specific  simulated  rendezvous  operations  with  local 
forces  (e.g.,  deliver  medical  supplies  to  local  clinic)  while 


aiming  to  maintain  a  presence  patrol  in  the  sector.  At  each 
checkpoint,  no  squad  dismounts  were  simulated  in  the 
environment;  instead,  the  Commander  just  communicated 
with  headquarters  as  if  the  dismount  activity  had  occurred. 
There  was  no  mandatory  wait  time  at  the  checkpoints,  so  the 
Commanders  continued  on  to  the  next  checkpoint  as  quickly 
as  they  desired.  After  completing  the  city  mission,  the  pair 
drove  back  to  the  FOB  to  complete  the  mission. 


Figure  3:  Vehicle  Driver  sitting  at  the  static  driver  station, 
talking  to  the  Commander  over  a  headset  and  controlling  the 

movements  of  the  RMS  with  a  yoked  steering  wheel.  Photo 
provided  by  Detroit  Arsenal  Media  Services. 

The  experimental  emphasis  was  on  the  Commander  and 
his  numerous  tasks,  including  route  planning  and  navigation, 
responding  to  various  auditory  communications  about 
mission  status  and  coordination,  and  maintaining  local 
situational  awareness  (LSA)  to  visually  detect  and  identify 
specific  individuals.  In  a  patrol  mission,  these  tasks  often 
occur  simultaneously,  especially  with  a  live  driver  who  is 
dependent  on  the  Commander  for  navigation  directions  and 
wayfinding  information,  since  the  Driver  had  only  a  60 
degree  field  of  view  and  no  map  of  the  Commander’s 
planned  route.  As  experimenters,  we  faced  the  challenge  of 
how  to  implement  each  of  these  tasks  in  a  realistic  fashion 
given  the  technical  constraints  of  the  available  simulation 
environment.  The  task  implementation  also  had  to  preserve 
enough  experimental  control  to  conduct  an  analysis  of  the 
EEG  data. 

Experimental  Questions 

The  main  experimental  objective  was  to  acquire  neural 
data  for  these  Commander  tasks  with  as  realistic  task  loading 
and  environment  interactions  as  possible  within  the 
simulation.  We  will  highlight  three  of  our  experimental 
questions,  and  each  will  be  discussed  in  more  detail  in  the 
Three  Example  Pathways  discussion  in  the  MBS  Data 
Analysis  section. 

First,  although  the  mission  tasks  have  dynamic  and 
variable  overlap  with  one  another,  can  the  neural  data  reveal 
timeframes  when  the  Commander  is  busy  with  many  tasks 
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compared  to  times  when  he  is  more  idle  with  less  frequent 
task  demands?  The  design  facilitated  this  analysis  by 
varying  the  task  difficulty  during  two  fixed  sections  of  each 
mission.  The  drive  between  the  FOB  and  the  city  at  the 
beginning  and  end  of  the  mission  (4  minutes  each  way)  was 
intentionally  quiet  with  only  2-3  mission  queries  from  the 
TOC  and  minimal  navigation  oversight  for  the  Driver  since 
the  outskirts  route  was  learned  easily  by  all  participants. 
Meanwhile,  the  mission  in  the  city  was  busy  with  consistent 
queries  from  the  TOC,  constant  chatter  on  the  radio  network 
among  other  platoons,  navigation  oversight  required  for  the 
Driver,  dynamic  traffic  and  obstacle  challenges  along  the 
mission  route,  and  time  pressure  to  arrive  at  the  rendezvous 
checkpoints  at  a  specified  time.  Here,  the  question  is  not 
what  task  the  Commander  is  performing,  but  whether  he  is 
completing  many  tasks  or  just  a  few.  If  this  neural 
information  can  be  identified  despite  the  movement  artifacts 
and  differences  in  the  task-specific  brain  processing,  a  BCIT 
may  be  designed  to  optimize  Soldier  performance  during 
times  of  potential  task  overload  (see  Pathway  Example  1). 

Second,  can  the  neural  data  capture  when  the  Commander 
listens  to  a  mission-relevant  compared  to  a  mission- 
irrelevant  auditory  communication?  The  design  used  both 
pre-recorded  TOC  communication  to  the  Commander  as 
well  as  pre-recorded  communication  among  members  of  a 
complementary,  but  not  mission-relevant,  platoon.  All 
communications  to  the  Commander  required  a  verbal  and/or 
motor  response.  Accuracy  in  these  responses  indicates 
whether  the  communication  was  successfully 
comprehended.  If  neural  measures  of  auditory 
comprehension  can  be  identified  in  complex  mission 
environments,  a  BCIT  may  be  developed  to  adapt  the 
interface  for  efficient  behavioral  responses  (see  Pathway 
Example  2). 

Third,  can  the  neural  data  indicate  detection  of  a  visual 
target?  The  Commander  was  responsible  for  detecting  and 
reporting  the  locations  of  the  local  coalition  forces  within  the 
city.  Similar  to  the  second  question  about  detecting  auditory 
communications,  neural  markers  for  visual  target  detection 
would  enable  a  BCIT  to  facilitate  the  supporting  detection 
tasks  and  adapt  the  interface  for  efficient  behavioral 
responses  (see  Pathway  Example  3). 

In  the  next  section,  we  will  describe  how  we  overcame 
some  of  the  challenges  for  task  implementation  to  achieve 
the  three  MBS  aims  during  mission  scenario  development. 

MBS  EXPERIMENTAL  DESIGN  CHALLENGES 

As  discussed  earlier,  an  MBS  design  has  three  main  aims: 
1)  to  situate  experiments  within  a  more  realistic  mission 
context;  2)  to  incorporate  tasks,  task  loadings,  and 
environmental  interactions  that  are  consistent  with  the 
mission’s  operational  context;  and  3)  to  permit  multiple 
sequences  of  actions/tasks  to  complete  mission  objectives 


[1],  Each  of  these  aims  had  practical  implications  for  the 
experimental  design. 

MBS  Aim  1:  Situate  in  mission  context 

The  first  MBS  design  aim  is  to  establish  a  mission 
framework  to  obtain  neural  data  that  is  obtained  in  a  context 
relevant  for  an  intended  BCIT  design.  For  this  experiment, 
the  chosen  mission  context  was  low-threat  security  patrols. 
Therefore,  a  specific  joint  operation  with  the  local  forces 
was  specified  for  each  of  the  six  missions,  and  the  TOC 
communications  for  each  checkpoint  were  in  line  with  this 
overall  mission  objective.  For  example,  one  mission  was  a 
Knock  and  Treat  operation  aimed  to  provide  medical 
treatment  to  local  civilians,  and  a  simulated  operation  within 
this  mission  for  the  dismount  squad  at  a  checkpoint  was  to 
deliver  an  injured  child  to  a  local  hospital.  Another  mission 
was  a  Snap  Vehicle  Checkpoint  operation,  where  one  of  the 
simulated  checkpoint  operations  delivered  the  necessary 
supplies  for  the  snap  VCP.  By  situating  each  mission  with  a 
low-threat  operation,  the  experimental  design  facilitates  an 
immersive  environment  for  the  Soldier  participant  to 
conduct  Commander  tasks  in  an  environment  that  matches 
his  expectation  from  his  experience  in  theatre.  The  match 
between  the  simulated  tasks  and  his  experience-based 
expectations  fosters  behavioral  actions  and  neural  responses 
that  are  as  realistic  as  possible  given  the  necessary 
constraints  for  a  neuroscience-based  analysis  of  task 
performance. 

MBS  Aim  2:  Appropriate  tasks  &  interactions 

The  second  MBS  design  aim  strives  to  link  the 
implementation  of  the  tasks  based  on  the  situated  mission 
context.  For  this  experiment,  this  critical  link  was 
implementing  the  expected  Commander  tasks  while 
minimizing  tasks  that  would  require  engagement,  since  the 
simulation  environment  used  for  this  experiment  did  not 
support  combat-related  behaviors  (e.g.,  vehicle  firing 
capability,  appropriate  attacking  or  fleeing  behaviors  for 
human  entities,  etc).  In  order  to  minimize  immersion¬ 
breaking  events  [4],  the  simulated  tasks  must  match  the 
capabilities  of  the  simulation  environment  with  the  expected 
participant  responses.  One  salient  challenge  for 
implementing  this  aim  was  identifying  a  low-threat,  visual 
target  detection  task. 

A  large  part  of  a  Commander’s  job  on  a  security  patrol  is 
to  maintain  local  situational  awareness  (LSA),  and  our 
mission-based  scenario  needed  to  include  a  target  detection 
task  to  incorporate  this  component.  To  avoid  engagement 
requirements,  the  task  could  not  place  human  threats  in  the 
environment  to  detect  since  the  Commander  would  not  be 
able  to  respond  and  engage  the  threat  in  appropriate  and 
expected  ways.  Likewise,  we  could  not  place  lEDs  in  the 
environment  for  the  Commander  to  find  since  the  vehicle 


Mission-Based  Scenario  Research:  Experimental  Design  and  Analysis,  Vettel,  et  al. 

Page  5  of  9 


UN  CL  AS  SIF IED//DIST  RIB  UTION  A.  APPROVED  FOR  PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


Proceedings  of  the  2011  Ground  Vehicle  Systems  Engineering  and  Technology  Symposium  (GVSETS) 


team  would  not  just  report  the  1ED  and  then  drive  by  it  as  an 
appropriate,  realistic  response.  We  settled  on  a  task  to  report 
the  location  of  local  coalition  forces,  explaining  the 
operational  relevance  of  ensuring  that  the  local  forces  were 
out  on  patrol  as  part  of  shared  security  measures  for  both 
local  and  US  forces.  The  reporting  requirement  for  this  task 
also  provided  a  mechanism  to  quantify  the  Commander 
performance  for  analysis  purposes  (see  Pathway  Example  3). 

In  addition  to  reporting  local  Soldiers  on  patrol,  a 
Commander  also  had  a  dynamic  visual  target  detection  task 
where  the  TOC  identified  a  new  target  to  find  after  the 
mission  began.  The  motivation  here  was  to  increase  the  LSA 
responsibilities  of  the  Commander,  allowing  him  to  feel  the 
more  realistic  sensation  of  several  tasks  building  up  as  he 
executed  his  planned  route  through  the  city  to  reach  his 
checkpoints  at  the  assigned  mission  time.  This  task  also 
increased  the  intensity  and  frequency  of  the  tasks  within  the 
city,  which  was  essential  for  the  experimental  question  about 
identifying  neural  patterns  indicative  of  variable  task  loads 
on  the  Commander  throughout  the  mission  (see  Pathway 
Example  1). 

The  selection  of  these  visual  target  detection  tasks 
highlight  the  influence  that  the  MBS  aim  to  enable 
appropriate  tasks  and  environment  interactions  had  on  this 
experiment  design.  The  targets  had  to  fit  the  low-threat 
mission  profile  to  eliminate  the  need  for  engagement,  but  the 
target  task  needed  to  be  demanding  with  a  challenging 
number  of  targets.  The  reporting  task  for  all  targets  provided 
the  necessary  metrics  for  analysis. 

MBS  Aim  3:  Enable  participant-specific  actions 

The  third  MBS  aim  addresses  the  desire  for  an  interactive 
experience  between  the  participant  and  the  simulation 
environment.  For  this  experiment,  the  challenge  arose  when 
balancing  a  Commander’s  route  planning  task  with  the  two 
visual  target  detection  tasks  and  the  timing  of  mission¬ 
relevant  auditory  communications  from  the  TOC. 

Before  each  of  the  six  missions,  the  Commander  planned  a 
route  through  the  city  to  the  three  checkpoint  locations  that 
ensured  he  arrived  at  the  checkpoints  at  the  assigned  mission 
times,  while  still  patrolling  enough  of  the  city  to  maintain  a 
presence  patrol.  This  meant  that,  as  experimenters,  we  did 
not  know  which  roads  in  the  city  would  be  traversed  during 
any  particular  mission;  however,  during  scenario 
development  for  the  experiment,  every  visual  target  had  to 
be  assigned  a  specific  location  in  the  environment.  Likewise, 
each  audio  communication  from  the  TOC  had  to  be  timed  to 
play  in  conjunction  with  particular  events  in  the  mission 
scenario.  In  short,  the  route  planning  task  implemented  an 
expected  task  for  the  Commander,  but  it  also  created  a 
challenge  for  target  placement  and  communication  timing. 

For  the  visual  targets,  our  solution  emphasized  the  most 
likely  routes  through  the  city.  In  particular,  more  targets 


were  placed  along  the  major  streets  in  the  city  since  most 
routes  would  have  to  incorporate  sections  of  these  streets.  In 
addition,  some  targets  were  placed  in  the  vicinity  of  the 
checkpoints  since  all  Commanders  would  traverse  these 
streets  as  part  of  the  mission  requirements. 

We  used  two  alternative  approaches  for  ensuring  mission- 
appropriate  timing  for  the  TOC’s  audio  communications  - 
vehicle  trip  lines  in  the  environment  and  a  live  experimenter 
who  was  able  to  play  particular  auditory  files  with  a 
soundboard.  Each  of  these  decisions  still  resulted  in  some 
minor,  unsolved  design  challenges.  The  trip  lines  for  the 
radio  change  were  triggered  anytime  the  vehicle  headed 
down  that  road,  even  when  the  traversal  was  for  presence 
patrol  and  not  checkpoint  arrival.  The  soundboard  ensured 
that  the  checkpoint  arrival  and  departure  messages  did  not 
face  this  same  challenge,  but  it  required  human  intervention 
which  is  not  preferred.  Nonetheless,  although  not  perfect,  the 
vehicle  trip  lines  and  soundboard  did  ensure  that  all 
Commanders  heard  the  required  communications  and 
supported  the  realism  of  the  TOC  interactions  expected 
during  a  security  patrol. 

The  third  MBS  aim  to  enable  participant-specific  actions 
within  the  simulated  mission  environment  provides  a  host  of 
challenges  for  the  experimental  design,  highlighted  here 
with  a  discussion  of  target  placement  and  communication 
timing.  The  intended  development  of  an  interactive  narrative 
algorithm  will  certainly  address  some  of  these 
implementation  challenges  [1],  but  managing  the  competing 
interests  of  enabling  participant  control  while  maintaining 
sufficient  experimental  precision  will  be  an  on-going 
challenge  for  MBS  experimentation. 

MBS  DATA  ANALYSIS 

The  increased  complexity  of  the  experimental  environment 
utilized  in  this  MBS  experimental  design  requires  different 
data  analysis  methods  and  data  interpretation  than  a  standard 
laboratory  experiment. 

An  event-related  potential  (ERP)  analysis  is  a  common 
approach  for  laboratory  experiments.  An  ERP  is  calculated 
by  segmenting  the  EEG  signal  into  epochs  of  time 
surrounding  the  event  of  interest,  such  as  the  appearance  of  a 
specific  sensory  stimulus,  and  then  averaging  time  epochs  of 
this  repeated  stimulus  type  to  reveal  a  common  EEG 
waveform.  Based  on  the  precise  timing  of  a  single  event,  the 
ERP  is  likely  to  indicate  brain  processing  related  to  that 
event  of  interest.  With  the  increased  complexity  of  this 
experiment,  where  different  event  types  often  occurred 
simultaneously,  it  is  difficult  to  use  this  experimental 
methodology  to  study  the  brain  signatures  underlying 
specific  event  types.  Critically,  our  experimental  objective  is 
not  to  describe  the  neural  processing  for  a  particular 
Commander  task;  instead,  we  aim  to  investigate  whether  any 
neural  signature  of  task-relevant  behavior  could  be  identified 
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despite  the  complexity  of  a  mission  with  concurrent  tasks 
and  noise  artifacts  introduced  by  less  constrained 
movements  of  a  Commander’s  head,  torso,  and  hands  [8]. 

Consider  an  analysis  of  brain  data  from  this  experiment 
that  seeks  neural  patterns  indicative  of  successful  visual 
target  detection.  Essential  to  this  analysis  is  the  ability  to 
know  whether  a  given  target,  i.e.,  a  coalition  Soldier,  is 
detected  and  when  this  detection  occurred.  While  the 
Commander  reports  when  they  see  a  target,  this  response 
does  not  indicate  the  time  of  detection.  First,  the 
Commander  was  performing  multiple  tasks  at  once,  so  the 
response  time  of  the  verbal  report  was  often  more  correlated 
with  the  number  of  simultaneous  tasks  being  performed  than 
it  was  with  the  precise  time  of  the  target  appearing.  Second, 
the  targets  did  not  suddenly  appear  on  the  Commander’s 
screen,  but  due  to  visual  perspective,  appeared  off  in  the 
distance  and  grew  as  the  Commander  approached  them.  This 
change  in  perspective  makes  it  difficult  to  code  an  algorithm 
for  when  the  target  was  detectable  to  the  human  eye.  Third, 
the  Commander  manipulated  camera  sensors  of  the 
environment,  so  how  much  of  the  environment  was 
processed  (or  what  entities  were  detected)  during  frequent 
sensor  changes  is  hard  to  quantify.  In  short,  establishing  a 
precise  time  that  a  target  was  detected  is  non-trivial. 

We  are  currently  exploring  several  methods  for  addressing 
this  event  timing  issue.  First,  we  are  using  pattern 
classification  methods  to  determine  whether  it  is  possible  to 
identify  brain  states  related  to  the  visual  target  processing  in 
time  windows  occurring  around  the  onset  of  the  visual 
target,  rather  than  a  precise  detection  time  within  a  few 
milliseconds  for  an  ERP  analysis.  This  approach  to  data 
analysis  is  useful  as  a  pragmatic  approach  to  developing 
BCITs,  but  provides  limited  useful  information  about 
underlying  brain  processing  for  specific  tasks  in  real-world 
environments  since  targets  in  our  MBS  experiment  co-occur 
with  other  events.  This  hinders  our  ability  to  determine  what 
neural  patterns  relate  to  the  target  detection  event  and  what 
relates  to  the  co-occurring  event(s).  It  does  not  preclude, 
however,  identifying  features  of  a  brain  state  that  are 
predictive  for  a  specific  task.  These  features  are  one  critical 
element  for  developing  a  BCIT. 

In  addition  to  pattern  classification  techniques,  we  are  also 
exploring  combining  eye-tracking  information  to  augment 
the  timing  accuracy  of  a  visual  target  detection  event  [9]  by 
using  the  time  an  eye  gaze  locks  the  target  entity  as  a  proxy 
for  the  time  the  target  was  detectable  by  the  participant. 
Finally,  we  are  examining  bottom-lip,  data-driven  methods 
to  detect  similarities  in  brain  signals  detected  throughout  the 
experiment,  and  check  their  correlations  with  experimental 
stimuli  in  order  to  identify  relevant  brain  signals  for  further 
processing. 

Through  the  use  of  these  and  other  techniques  [10],  we  aim 
to  reveal  predictive  features  in  the  EEG  signal  for  a  brain 


state  of  interest.  These  features  can  then  feed  the 
development  of  a  BCIT  to  detect  brain  states  that  indicate 
time  frames  when  a  predetermined  change  to  the  system 
interface  could  optimize  performance. 

THREE  EXAMPLE  PATHWAYS:  MBS  TO  BCIT 

Thus  far,  this  paper  has  only  discussed  the  challenges 
when  implementing  an  MBS  study.  This  section  describes 
the  expected  Army  relevant  capabilities  afforded  by  MBS 
experimentation  by  discussing  three  example  pathways  of 
how  an  MBS  analysis  could  lead  to  the  development  of  a 
BCIT.  These  three  pathways  link  back  to  the  three 
experimental  questions  highlighted  in  the  section  describing 
the  Mission-Based  Scenario  experiment. 

Pathway  Example  1:  Commander  Taskload 

During  each  mission,  the  number  and  frequency  of  the 
Commander’s  tasks  varied  during  different  segments  of  the 
mission.  The  drive  between  the  FOB  and  the  city  at  the 
beginning  and  end  of  the  mission  had  infrequent  tasks,  while 
the  mission  to  the  three  checkpoints  in  the  city  had  many 
tasks  occurring  simultaneously,  all  competing  for  the 
Commander’s  time  and  attention.  This  design  enables  an 
analysis  to  identify  neural  signatures  of  taskload  intensity. 

This  analysis  could  lead  to  a  BCIT  that  can  mitigate  task 
overload  during  times  of  peak  intensity.  The  appropriate 
mitigation  strategy  would  need  to  be  identified 
experimentally,  but  some  potential  mitigation  strategies  may 
include  supporting  a  simplified  version  of  the  interface.  This 
simplified  interface  would  minimize  the  visual  complexity 
of  the  screen  (decreasing  sensory  overload)  while  preserving 
the  essential  elements  for  the  critical  mission  tasks.  If  the 
system  is  set  up  for  alerts  on  both  critical  and  non-critical 
mission  elements,  the  system  could  revert  to  critical 
information  only.  A  crewstation  contains  many  systems  with 
many  levels  of  alerts  and  system  information,  so  identifying 
times  when  the  crewstation  may  be  too  overwhelming  based 
on  the  Commander’s  taskload  could  serve  to  improve 
mission  performance. 

Pathway  Example  2:  Communication  Actions 

During  each  of  the  six  missions,  the  TOC  communications 
about  the  mission  required  just  a  verbal  response  or  a  verbal 
response  plus  a  touchscreen  button  response.  This  design 
facilitates  a  pattern  classification  analysis  to  investigate  if 
EEG  signal  features  can  indicate  when  the  communication 
requires  a  verbal-only  response  versus  communications  that 
require  both  verbal  and  button  response. 

If  these  two  response  states  can  be  differentiated,  the 
system  could  dynamically  alter  the  size  of  the  interface 
buttons  to  make  it  easier  for  the  Commander  to  touch  the 
correct  button  despite  the  jostling  of  the  body  due  to  the 
motion  of  the  vehicle  over  terrain.  This  dynamic  interface 
adaptation  may  improve  the  response  time  and/or  the 
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accuracy  for  that  task  by  increasing  the  ease  of  interaction 
between  the  Commander  and  the  interface. 

Pathway  Example  3:  Visual  Target  Reporting 

In  this  experiment,  the  Commander  was  responsible  for 
reporting  the  location  of  visual  targets  in  the  local 
environment.  Using  an  ERP  approach,  neuroscience  research 
in  the  laboratory  has  identified  a  stereotypic  response  in  the 
EEG  signal,  known  as  the  P300,  which  reflects  the  detection 
of  a  target  with  a  positive  deflection  of  the  wave 
approximately  300  milliseconds  after  the  target  stimulus  is 
detected  [11].  Pattern  classification  approaches  may  identify 
a  neural  signature  that  captures  this  marker  of  the  target 
detection  process  without  requiring  a  time-locked  signal 
average  (i.e.,  ERP).  Thus,  a  classification  analysis  of  the 
general  timeframe  of  the  visual  target  detection  events  in  this 
dataset  would  reveal  whether  a  neural  signature  for  the 
detection  event  is  discernable  despite  the  increase  in  task 
complexity  and  movement  artifacts  and  decrease  in  the 
precision  of  the  event  timing. 

A  common  mission  task  is  to  mark  a  digital  map  of  the 
sector  with  the  location  of  identified  targets,  although  that 
was  not  a  task  we  incorporated  based  on  our  decision  to 
simulate  low-threat  security  patrols.  For  missions  where  a 
map  of  detected  targets  is  critical,  the  efficiency  of  the 
markup  may  be  improved  by  linking  a  neural  response  to  a 
target  to  the  fixation  point  of  the  Commander’s  eyes  as  an 
estimate  of  where  to  place  the  target  on  a  digital  map  of  the 
environment.  The  system  would  link  the  gaze  point  on  an 
indirect  vision  camera  system  to  a  grid  location  that  links  the 
current  zoom/ focus  on  the  vehicle  camera  to  a  point  in  the 
mission  map  of  the  patrol  sector.  The  system  could  then 
prompt  the  Commander  to  accept,  modify,  or  reject  the 
target  placement.  This  type  of  BCIT  may  optimize  target 
reporting.  Further  discussion  of  this  idea  can  be  found  in 
[12]. 

Future  Pathways 

These  three  examples  highlight  a  sample  of  the  possible 
BCIT -based  capabilities,  but  they  are  not  an  exhaustive  list. 
This  experiment  is  part  of  an  initial  effort  at  implementing  a 
neuroimaging  experiment  that  employs  an  MBS  design  and 
analysis.  Each  will  continue  to  improve  over  time, 
identifying  pathways  to  transition  MBS  analyses  to  a  BCIT 
within  a  definable  timeframe  for  particular  performance 
enhancements  and  enable  Army-relevant  capabilities. 

SUMMARY 

The  overarching  goal  of  an  MBS  design  is  to  enable  the 
development  of  a  BCIT  that  can  leverage  measures  of  brain 
activity  to  enhance  the  interactions  between  Soldiers  and  the 
systems  required  to  execute  their  missions.  To  optimize 
BCIT  development,  we  argue  that  the  experiment  must 


match  as  closely  as  possible  the  targeted  environment  for 
implementation,  and  we  propose  two  specific  elements  to 
accomplish  this  aim. 

The  first  element  is  the  simulated  environment,  using  a 
closed-hatch  crewstation  environment  as  a  critical  venue  to 
transition  neuroscience  research  from  constrained  laboratory 
settings  to  unconstrained,  real-world  environments.  The 
second  element  is  the  MBS  design  and  analysis  framework. 
The  three  aims  of  an  MBS  design  capture  the  relevant 
contextual  effects  on  the  Soldier’s  mission  tasks,  actions, 
and  environmental  interactions  in  order  to  understand  how 
the  brain  functions  in  complex,  real-world  environments. 
The  MBS  analysis  approaches  emphasize  methods  to 
identify  predictive  neural  signatures  for  brain  states  relevant 
for  task  performance.  By  incorporating  these  two  elements, 
neuroscience  experiments  hold  promise  for  enhancing  Army 
capabilities  by  using  BCITs  to  improve  Soldier-system 
performance. 

MBS  scenario  development  marks  a  shift  in  experimental 
design  from  earlier  EEG  studies  conducted  on  the  Ride 
Motion  Simulator.  Prior  experiments  have  used  fixed  routes 
with  only  one  participant,  where  a  solo  Commander  is 
driven  by  the  simulation  through  the  patrol  mission.  This 
fixed  route  ensures  that  all  targets  placed  are  seen,  and  any 
communications  that  require  precise  timing  within  the 
mission  scenario  can  be  scripted  easily.  While  this  more 
controlled  design  facilitates  traditional  neuroscience-based 
analyses,  it  fails  to  capture  the  contextual  effects  on  the 
participant’s  tasks,  actions,  and  environmental  interactions. 
It  is  not  clear  that  our  brains  will  perform  tasks  identically  in 
a  simplified  environment  (fixed  route,  single  rather  than 
concurrent  tasks,  etc)  and  in  a  dynamic,  complex 
environment  (route  navigation  and  alterations,  concurrent 
tasks,  etc).  Consequently,  the  design  challenges  tackled 
during  our  first  mission-based  scenario  experiment  on  the 
RMS  seem  necessary  to  capture  a  Commander’s  dynamic 
mission  behavior.  While  design  enhancements  are  targeted 
for  future  MBS  studies,  this  experiment  provides  our  first 
neuroimaging  dataset  on  the  RMS  that  captures  dynamic 
Commander-Driver  teaming  during  a  more  realistic 
simulation  of  a  low-threat  security  patrol  mission.  We  argue 
that  these  types  of  datasets  will  enable  a  pathway  between 
simulation  experiments  and  BCIT  development  that  reveal 
the  promise  of  neuroscience  research  for  the  Army. 
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